Prediction from Blog Data

We have approximately one year’s worth of blog posts from [1] with over 12 million web blogs tracked. On average there are 500 thousand blog posts per day. In this project, we are attempting to extract from the blog data the set of features that are predictive of the movie gross sales, critics ratings, and viewers ratings (collected by sites like [3]). Having gained this insight, we are applying machine learning techniques to make predictions on sales and ratings for future movies. This is useful for example, to see if the ‘buzz’ surrounding a movie is sufficient to obtain high sales, or whether additional marketing is required. In addition, we could automatically figure out ratings of movies based on their mentions in blogs. This could be useful for movie rating sites like Rotten Tomatoes [3] and IMDB [4], either to verify the correctness of ratings, or to ‘seed’ ratings for a new movie or a movie that did not exist in their database. Additionally, in some instances information like gross sales may not be publicly available, so we would like to make a prediction instead. At a high level, we would like to see if the “online chatter” is useful for analysis and prediction of the quality and popularity of various items, be it movies, songs, books, restaurants or other commercial products. In our project, for the top movies of 2008 (≈ 200 of them that have non-negligible blog mentions) and top movies of all time (≈ 250), we try to predict the following: the gross sales, the critics rating and the average viewers rating. We selected a long list of relevant features and populated them after parsing the blog data. After performing training, crossvalidation, feature and model selection with a primitive set of models, we have reasonable error rates for most output variables. We find that Naive Bayes and SVM are the best prediction algorithms for our data, and PCA generally works best as a feature selection method. Finally, we have observed some interesting patterns that suggest that quite accurate predictions can take place, given more sophisticated algorithms.